DOCUMENT RESUME 






ED 424 502 



CG 028 781 



AUTHOR 
TITLE 
PUB DATE 
NOTE 



PUB TYPE 
EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



Brammer , Robert 

Effects of Experience and Training on Diagnostic Accuracy. 
1998-08-00 

15p.; Paper presented at the American Psychological 
Association Annual Convention (106th, San Francisco, CA, 
August 14-18, 1998) . 

Reports - Research (143) -- Speeches/Meeting Papers (150) 

MFOl/PCOl Plus Postage. 

Artificial Intelligence; *Clinical Diagnosis; Clinical 
Experience; Clinical Psychology; * Information Seeking; 
♦Professional Training; *Psychologists ; Structural Equation 
Models; *Work Experience 
Expertise 



ABSTRACT 



The interview process was studied to uncover the 
relationship of expertise in psychotherapy to the likelihood of accurate 
diagnosis. Experience and training affect the number of diagnostic questions 
clinicians ask as compared to personal, family, social, occupational, and 
history questions; and this in turn affects the accuracy of the diagnoses 
offered. Psychologists and psychology graduate students (N=138) interacted 
with an artificial-intelligence program that simulated a date-rape client's 
responses to the questions they asked. Participants were asked to conduct a 
"clinical interview" by typing in their questions. At the end of their 
interview, the participants provided a brief diagnosis for the client. A path 
analysis revealed that clinical experience is a strong predictor of the 
ability to form an accurate diagnosis, but an individual's level of training 
affects diagnostic accuracy indirectly. Training and experience do not appear 
to help psychologists accurately diagnose a client's condition when they are 
provided limited information about the client; however, participants with 
higher levels of training asked more diagnostic questions, which in turn 
helped them to derive accurate diagnoses. (Author/EMK) 
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Abstract 

Psychologists and psychology students (n = 138) were provided an artificial- intelligence 
program that simulated a date-rape client’s responses to the questions they asked. Participants 
were asked to conduct a “clinical interview” by typing in their questions. At the end of their 
interview, the participants provided a brief diagnosis for the client. A path analysis revealed that 
clinical experience is a strong predictor of the ability to form an accurate diagnosis, but an 
individual’s level of training affects diagnostic accuracy indirectly. Participants with higher 
levels of training asked more diagnostic questions, which, in turn, helped them to derive accurate 
diagnoses. 
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Psychologists interested in the nature of expertise have focused on how experts across 
different fields use training and experience to increase the likelihood of desired outcomes 
(Bloom, 1985; Cooke, 1992; de Groot, 1978; Ericsson & Crutcher, 1990, Ericsson & Chamess 
1994; Ericsson, Krampe, & Tesch-Romer, 1993; Hayes, 1981; Patel & Groen, 1988). Despite the 
interest in other fields, little research exists concerning the process of making a psychological 
diagnosis (Etringer, Hillerbrand & Claibom, 1995). 

As Goodyear (1997) notes, experience is often used as a proxy for expertise. Ericsson and 
Lehmann’s (1996) argument that 10 years are required for the development of expertise in any 
field continues to champion the correlation of experience with expertise. The therapist, however, 
must deliberately attempt to develop skills for experience to have the desired effect on expertise 
(Clark, 1992; Ericsson et al., 1993). Previous research has shed little light on whether these 
processes exist in developing diagnostic expertise (Gelso & Fretz, 1992) 

Most research on the relationship between training and the diagnostic process has focused 
on the ability of experts to form accurate diagnoses (e.g., Christensen & Jacobsen, 1994; Dawes, 
1994; and Goldberg, 1959, 1968; Hattie, Sharpley, & Rogers, 1984; and Stein & Lambert 1989, 
1995). This contention has received support from Smith and Glass (1977), whose meta-analysis 
on the effectiveness of psychotherapy revealed that credentials (e.g., Ph.D., M.D., no advanced 
degree, etc.) make no statistically significant impact. At best, the current body of literature 
supports the contention that professionals have roughly the same client outcomes as 
paraprofessionals (Berman & Norton, 1985). 

However, the inability of trained psychologists to outperform minimally trained 
nonpsychologists in diagnosis may only when all parties have access to the same information. 
The interview process itself might reveal information necessary to make accurate diagnoses 
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(Gelso, 1992). Do therapists with doctoral degrees emphasize these issues more than 
psychologists in training? 

Apparently, therapist confidence is a confounding factor in the process of diagnostic 
judgment. Therapists typically report high levels of confidence in their judgments, but Vallone, 
Griffin, Lin, and Ross (1990) reported that therapists’ levels of confidence are inversely 
correlated with accuracy (see also Arnold, 1993). As psychologists become less accurate, they 
evoke high manifest levels of confidence (Etringer, Hillerbrand, & Claibom, 1990). There is a 
long history of literature indicating that therapists report higher levels of confidence than they 
merit (Dumont, 1993; Dunning, Griffin, Milojkovic, & Ross, 1990; and Glidewell & Livert, 
1992). 

Despite their problem with overconfidence, experienced therapists are better at describing 
which of their judgments are more likely to be accurate (Garb, 1989, 1996). Others have noted 
that confidence changes with information. Oskamp (1982) found that clinicians’ confidence in 
their judgments grows with additional knowledge of the client. Hillerbrand and Claibom (1990) 
also noted that experts feel more knowledgeable and confident in their judgments even though 
their cognitive processing does not differ significantly firom those of novices. 

In short, training and experience do not appear to help psychologists accurately diagnosis 
a client’s condition when they are provided limited information about the client. However, there 
is little research investigating whether training and experience might assist psychologists in 
seeking information that may lead to a more accurate diagnosis. This study investigates whether 
experienced psychologists seek different information than novices and whether they use this 
information to arrive at more accurate diagnoses. 
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Method 



Participants 

Participants were 138 (86 men and 52 women) psychologists or psychology students. 
Most (n = 68; 49%) had a doctorate in psychology or had been advanced to candidacy in a 
doctoral program (n = 14, 10%). The rest (n = 56; 41%) ranged from those who were just 
beginning graduate training to doctoral candidates. 

Materials 



The data was collected and initially reported by Brammer (1997). An interactive, 
artificially intelligent computer program, modeled after O’Byme’s (1993) study, was written for 
this study. The program approximates a real-life assessment situation by allowing participants to 
“interact” with the client. Initially, the participants read a one-page description of the client’s 
presenting problem: 

So you want to know about me. OK, my name is Chris. I'm a 20 year-old 
woman who works at a bookstore and attends college part time. I started college 
on the East Coast, but it didn't work out. I moved to California two years ago. 

I guess I am here because - because (she starts to cry) I feel so dirty and 
violated. My ex-boyfiiend, Mark, raped me last weekend. I thought I could 
handle it on my own, but it's too much for me. (She begins to speak more rapidly). 

Dr. Smith recommended you and my roommate thought it might help to 
talk to someone. The hardest part is how my parents handled all of this. They 
became angry and said they weren't going to pay for my schooling anymore. 

They want me to move back home. I feel so confused — I need help. 

From here, participants could ask any question they wished. The computerized client had 

information on the etiology of several psychological problems: psychosocial history, 

developmental history, family background, and facts about the crisis. As the participant asked 

questions, the computerized client responded with a one-paragraph answer. For example, the 
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participant could ask, “I know this is difficult, but tell me what happened the night of the rape.” 

To this, the computer would respond with the statement: 

Mark and I went out three times. Once to the movies, once to dinner, and finally on a 
camping trip with his friends. We shared a tent, and I told him I didn't want to make love 
with him, but he hit me and tore off my panties. He must have sensed that I was attracted 
to him. It's my fault. 

Dependent Measures 

The programmed tracked the types of questions the participants asked. It was 
h 3 qjothesized that psychologists with more experience or those with doctoral-level training would 
ask fewer overall questions and fewer questions regarding the client’s 
personal/ family/social/occupational hfrtory. Instead, it was hypothesized they would ask 
diagnostically related questions, which would likely help them to accurately diagnose the client. 

In order to identify the client’s current diagnosis, all of the client’s pre-generated 
statements were sent to 4 psychologists, ranging in experience from 1 to 22 years. This provided 
them with an advantage over the participants in the study who only saw responses to their 
questions. The raters created diagnoses based on this information and obtained an interrater 
reliability of .87 using Cronbach’s alpha. The following diagnoses received at least three votes: 
Major Depression, Bipolar II Disorder, Alcohol Abuse, Acute Stress Disorder, Borderline 
Personality Disorder, and Nicotine Dependence. 

Four students from West Texas A&M University rated the participants’ responses. For 
each correct diagnosis, the participant received one point. After averaging scores received from 
the raters, participant scores ranged from 0 to 5. Interrater reliability for the student judges was 
.74 using Cronbach’s alpha. 
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Results 

A path analysis (using Statistica) was performed to examine whether training and 
experience contributed to the creation of an accurate diagnosis directly or whether diagnosis 
came because experienced and well-trained practitioners asked diagnostically related questions. 
The path analysis (see Figure 1) depicts the effects of the exogenous variables (level of academic 
training and years of experience) and the endogenous variables (the total number of questions 
asked, the number of personal/family/social/occupational history questions asked, and the 
number of questions directly connected to a possible diagnosis) on diagnosis accuracy. The most 
significant paths leading to the correct diagnosis were experience (direct effect) and a connection 
between training (doctoral level) and the number of diagnostic questions asked. The total number 
of questions and number of history questions asked were associated with experience but they 
were not associated with providing correct diagnoses. Overall, the model had a strong goodness 
of fit (x" = 82.490, p < .001). 
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Figure 1 : Path diagram representing a model of moderators of accurate psychological diagnoses. 
Note: ***=p<. 001 



Discussion 

Most of the research investigating mental health professionals’ diagnostic accuracy 
examined whether differences between professional and novices when they each had the same 
clinical information (e.g., Christensen & Jacobsen, 1994; Dawes, 1994; and Goldberg, 1959, 
1968; Hattie, Sharp ley, & Rogers, 1984; and Stein & Lambert 1989, 1995). Such studies may 
have overlooked a relationship between experience, training, and diagnosis because they failed to 
examine clinicians’ information gathering processes. 

In this study, while experience independently predicted the ability to form an accurate 
diagnosis, a participant’s level of training had no direct effect. Instead, doctoral level therapists 
were more likely to ask diagnostic questions, and by asking these questions, they came derived 
accurate diagnoses. Rather than follow the client’s prompts or emphasize history issues, doctoral 
level psychologists focused on the problems at hand such as suicide, depression, and alcohol. 
This developing inquiry skill allows therapists to exhibit diagnostic skills significantly beyond 
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their colleagues with less training or experience, which corresponds to Patel and Groan’s (1991) 
findings inquiry skills of medical students. 

The results for this study, however, should be viewed with some caution for several 
reasons. First, the medium of the computer may have changed the ways the participating 
psychologists may have handled the case presentation. Even if the types of questions asked were 
similar to those the participants would have asked a client, the time spent asking questions may 
have differed in this study. Participants may have enjoyed the process of taking as much time as 
they wished without being prompted, to continue. This freedom likely impacted the therapists’ 
inquiries. 

The automated client responses and lack of sensory input also may have changed the 
questions that were asked. Although the computer program was successful in answering 
questions, true dialogue was not possible. The therapists were also unable to react to client cues. 
The inability for the computer to produce tones, gestures, pauses, or other nonverbal cues may 
have changed the therapists’ mode of inquiry. 

Third, there is no way of knowing if participants would have reacted differently if the 
case were about a male client. Gender effects should be explored in future studies. Nevertheless, 
the results of this analysis provide an important first step to understanding the ways experience, 
academic training, cognitive complexity, and assessment task change psychologists’ modes of 
inquiry. 

Despite these weaknesses, this study provides increased depth and perspective to 
O’Byme’s (1993) and Brammer (1997) findings. The weaknesses discussed here do not appear 
significant enough to overcome the strong effect sizes found. The findings here suggest that 
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experience and training affect the number of diagnostic questions clinicians ask, and this, in turn, 
affects the accuracy of the diagnoses offered. 
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